[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, take the renderer's typed config directly - #43

## Summary

- keep the H100 README badge scoped to `branch=main`
- add a B200 integration-test badge scoped to `branch=main`
- run both H100 and B200 workflows from CIFlow tags and once daily at
midnight UTC
- remove the B200 push-to-main, path, and manual-dispatch triggers

## Why

The H100 badge filtered for `main`, but the workflow had no scheduled or
push-to-main runs and therefore no matching status. Daily scheduled runs
provide a consistent `main` status for both dedicated accelerator lanes,
while CIFlow tags retain on-demand testing without running B200 on every
matching push to `main`.

## Test plan

- `/data/users/jianiw/torchtitan/.venv/bin/pre-commit run --files
README.md .github/workflows/integration_test_h100.yaml
.github/workflows/integration_test_b200.yaml`
- verified the H100 and B200 workflow links resolve successfully

## Summary

Correct DeepSeek V4 model FLOPs estimation used by TorchTitan
metrics/MFU.

The estimator now follows the current TorchTitan model accounting
pattern:

- use `get_nparams_and_active_nparams()` for total and active MoE
parameters;
- count DeepSeek V4 sliding-window and compressed sparse attention from
the final layer configs;
- include the CSA indexer score contraction;
- include MTP attention plus repeated LM-head and multi-stream `h_proj`
work.

This keeps the calculation model- and backend-independent; it does not
encode NPU/AscendC implementation details.

## Validation

Adds a CPU-suite meta-device regression test for `deepseek_v4_flash`
with one MTP layer at sequence length 4096:

- total parameters: `290,942,278,866`
- model FLOPs/token: `92,762,352,876`

The test builds the complete model on `meta` and calls the same
`Model.Config.get_nparams_and_flops()` integration point used by
metrics.

---------

Co-authored-by: depeng <166494784+depeng1994@users.noreply.github.com>

## Summary

- Set eps=1e-6 directly on every DeepSeek RMSNorm config, matching the
DeepSeek V2/V3 reference configuration.
- Avoid post-construction config traversal; each norm now declares its
model-specific epsilon alongside its other parameters.
- Refresh the affected fake-PG and CP/PP loss and grad-norm baselines
using the full-precision values from CI artifacts.
- Rebase onto the latest main branch.

The shared RMSNorm default remains 1e-5, so other model families such as
Kimi are unchanged.

DeepSeek reference:
https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json

## Validation

- Pyrefly on torchtitan/models/deepseek_v3/__init__.py: 0 errors
- 31 relevant CPU unit tests passed
- All RMSNorm configs in debugmodel, 16B, 236B, and 671B use 1e-6,
including MTP configs
- Updated loss and grad-norm baselines exactly match the failed CI
TensorBoard artifacts
- Targeted pre-commit hooks passed (Pyrefly run separately)

Math-Verify would parse a response like this:
```
 " Final Answer:
  The last three digits of \(2003^{2002^{2001}}\) are:
  \[Answer: 009\]"
```

and try to calculate "(2003^{2002^{2001}", leading to a hang. In the
past, this was not an issue. We would just timeout. But currently, we
cannot timeout due to some monarch issues and math_verify not being in
the main thread.

This PR workaround is to improve the parser. Now we demand the answer to
be in `\boxed(...)`. I ran 50 steps without hanging.
https://meta.wandb.io/felipemello/titan_rl/runs/ef3zj2gs?nw=nwuserfelipemello

torch is removing torch/cuda/_annotate_cuda_graph_trace.py, the offline
joiner that merged CUDA-graph kernel annotations into an already-written
trace. Its replacement is a cuda_graph_annotations argument on
export_chrome_trace, which splices the annotations in during the export
pass, so the trace is never read back and re-written -- for the gzipped
traces the profiler has produced since
 pytorch#3483 that was paying the compression cost twice.

The trace handler now passes get_cudagraph_annotations() straight to
export_chrome_trace, and cudagraph_annotate_trace_post_processor goes
away with its import. Passing an empty mapping is defined to mean the
same as passing none, so the handler no longer needs the early return
the post-processor had.

The two post-processor tests are replaced by one that checks the handler
hands the captured annotations to the export of the .json.gz path. The
gzip round-trip case they covered is now torch's exporter picking
compression off the file suffix, tested there rather than here.

Depends on the torch-side change; until it lands in a nightly,
export_chrome_trace does not accept cuda_graph_annotations.

Authored with assistance from Claude Code.

## Summary

- Remove the in-tree `torchtitan.experiments.forge` implementation and
example.
- Remove its stale CODEOWNERS entry.
- Direct post-training users to the standalone
[TorchForge](https://github.com/meta-pytorch/torchforge) project.

This is a breaking change for users that import
`torchtitan.experiments.forge`
directly from TorchTitan `main`. Published TorchTitan releases are
unchanged.

## TorchForge validation
This validation confirms that TorchForge's core GRPO training path works
with
the released TorchTitan 0.2.0 package and PyTorch 2.9.0. It does not
claim that
every TorchForge workflow or the unmodified README installation was
validated.

I tested standalone TorchForge in a clean Python 3.12 uv environment on
an H100
with:
- `torch==2.9.0+cu128`
- `torchvision==0.24.0+cu128`
- `torchtitan==0.2.0`
- `vllm==0.13.0`
- `torchmonarch==0.2.0`

Results:
- The SFT and GRPO entry points imported successfully.
- All 311 TorchForge unit tests passed.
- A Qwen3-1.7B GRPO smoke run completed two training steps, including
  trainer-to-vLLM weight synchronization.

No TorchForge or TorchTitan source patches were needed. This was not a
fully
out-of-the-box dependency validation: local network restrictions
required a
local TorchStore checkout, optional metrics were disabled, and the GRPO
run used
a reduced two-step smoke configuration. The test therefore validates the
core
training and weight-synchronization path, not the exact README
installation.

Standalone TorchForge currently pins TorchTitan 0.2.0 and still consumes
the
Forge APIs shipped in that release. This test does not establish
compatibility
between TorchForge and TorchTitan `main` or 0.3.0.


### TorchTitan 0.3.0 compatibility

I also tested a Python 3.12 environment with `torch==2.14.0+cu130`,
`torchvision==0.29.0+cu130`, `torchao==0.18.0+cu130`, and
`torchtitan==0.3.0`. Current TorchForge is not compatible with this
release
stack:

- Its package metadata requires `torch==2.9.0` and `torchtitan==0.2.0`.
- The SFT entry point fails during import because TorchForge still
imports
  `torchtitan.experiments.forge.train_spec`, which is not present in
  TorchTitan 0.3.0.
- TorchForge also imports `torchtitan.experiments.forge.job_config`,
which was
  present in TorchTitan 0.2.0 but is not present in 0.3.0.

The failure occurs before GPU training begins. Supporting TorchTitan
0.3.0
requires changes in the standalone TorchForge project; no compatibility
patches
were applied for this test.

Co-authored-by: JessicaZhong <zhengjesszhong@gmail.com>

## Summary

- Add an opt-in Triton implementation of Qwen3.5 `OffsetRMSNorm`,
including fused forward, input-gradient, and weight-gradient kernels.
- Preserve the existing zero-centered parameterization, state-dict keys,
FP32 accumulation, DTensor sharding contracts, CPU fallback, and
`torch.compile` support.
- Add `rl_grpo_qwen3_6_27b_varlen_perf`, an 8-GPU Qwen3.6-27B RL config
that uses the Qwen3.5-compatible model flavor and enables the override
for both trainer and generator.

The standalone override can be enabled with:

```bash
--override.imports torchtitan.overrides.offset_rmsnorm.triton_offset_rmsnorm
```

The new RL config uses TP2 x FSDP2 for the trainer and TP4 for the
generator. Full activation checkpointing and BF16 Adam states keep the
27B trainer within H100 memory.

## Why

The stock operation computes `(1 + weight) * rmsnorm(input)` as separate
eager PyTorch operations. It materializes intermediates and launches
separate square, reduction, reciprocal-square-root, add, multiply, and
cast operations. The Triton path computes FP32 RMS statistics and the
offset scaling in one forward kernel. Its custom backward similarly
fuses the input-gradient work and reduces the weight gradient in Triton.

## Numerics and compatibility

- Forward, input gradient, and weight gradient are checked against both
an FP32 PyTorch reference and an FP64 golden reference.
- Coverage includes FP32, BF16, FP16, non-power-of-two dimensions, Qwen
hidden sizes 4096/5120, near-zero variance, and zero variance.
- `torch.library.opcheck` validates schema, FakeTensor, and autograd
registration.
- A fullgraph `torch.compile` forward/backward test passes.
- The override preserves checkpoint layout and applies to every Qwen3.5
`OffsetRMSNorm` config.
- The fused inference path successfully captures and replays inside vLLM
CUDA Graphs.

## Performance

All comparisons use the same code/configuration except for the override.

### Pretraining

Qwen3.5-27B, 4x H100, BF16, TP2 x FSDP2, sequence length 4096, 8192
unique tokens/step, full activation checkpointing, C4 packed text, and
BF16 Adam states. CUDA Graphs are disabled because variable
packed-document mask/GDN metadata currently changes structure between
batches. Statistics are steps 5-34 after warmup.

| Metric | Stock | Triton | Change |
| --- | ---: | ---: | ---: |
| Throughput/GPU | 1156.69 tok/s | 1235.30 tok/s | **+6.80%** |
| End-to-end step | 1.7707 s | 1.6583 s | **-6.35%** |
| MFU | 18.63% | 19.89% | **+1.27 pp** |
| Max reserved memory | 89.93 GiB | 89.93 GiB | unchanged |

A second paired short run measured +12.22% throughput; the table reports
the conservative longer run. Both 34-step C4 smoke runs followed the
same loss trend and ended at 3.2830 (stock) and 3.2800 (Triton).
Standard FP32 Adam states OOM while materializing optimizer state on
this topology, so both sides use `fused_opt_states_bf16`.

### Inference

Qwen3.5-compatible 27B model, 4x H100, TP4, BF16,
`torchtitan.experiments.rl.generate`, one sequence, 1024 generated
tokens, and vLLM `FULL_AND_PIECEWISE` CUDA Graphs with decode capture
size 1. The model was random-initialized to isolate execution
performance from the unrelated HF-to-TP4 checkpoint-loading path.

| Metric | Stock | Triton | Change |
| --- | ---: | ---: | ---: |
| Decode throughput | 51.814 tok/s | 60.227 tok/s | **+16.24%** |

## Validation

```text
pytest tests/unit_tests/cpu/test_triton_offset_rmsnorm_override.py -q
4 passed

pytest tests/unit_tests/gpu/test_triton_offset_rmsnorm_override.py -q
5 passed, 9 subtests passed

pytest torchtitan/experiments/rl/tests/test_generator.py -q
26 passed, 1 skipped

SKIP=pyrefly-check pre-commit run --all-files
all enabled hooks passed
```

The repository-wide Pyrefly hook was skipped locally because this
environment lacks the optional `torch_checkpointing` and `attn_gym`
packages and uses a PyTorch build with known API skew; the changed
Python files pass AST, flake8, ufmt, pydoclint, and codespell.

Summary:
Run retention only after the previous save has completed and before the
next save starts. `_purge_stale_checkpoints` receives the step being
saved so rank 0 can preserve the exact current published and staging
directory names even if another rank creates them while cleanup is
running. This avoids a cross-rank barrier in the checkpoint save path.

The method accepts an optional staging directory prefix and matches
those directories separately instead of teaching `_parse_step` about
backend-specific temporary names. Valid published checkpoints alone
participate in retention, and the manager keeps `keep_latest_k - 1` of
them to reserve a slot for the current save. Unknown directory names
remain untouched.

Abandoned directories from other steps use the existing purge thread.
The current step is excluded before paths are queued, so background
deletion cannot target the active save. The DCP manager moves its purge
call from after save dispatch to the settled window before dispatch and
passes the current step.

Test Plan:
`pytest tests/unit_tests/cpu/test_checkpoint.py
tests/unit_tests/cpu/test_torch_checkpointing.py -q`

64 passed.

Tests cover DCP purge ordering, current-save protection, retention slot
reservation, incomplete checkpoint cleanup through the purge thread, and
canonical parsing.

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/4197).
* pytorch#4457
* pytorch#4277
* pytorch#4279
* pytorch#4278
* pytorch#4191
* pytorch#4190
* pytorch#4189
* pytorch#4188
* __->__ pytorch#4197

Summary:
`TorchCheckpointingManager` has been config-only since pytorch#4058: it builds
its
backend and then raises `NotImplementedError` from every operation. This
implements the save path against that backend, leaving load for a later
change.

Saves follow the same cadence contract as the DCP manager -- interval,
first
step, and last step -- and hand the flattened state dict to the backend,
which
returns a future tracked in `save_future` and awaited by
`maybe_wait_for_saving`.
On a step where no save is due, the manager prewarms the backend's
staging
buffers once, so the first real save does not also pay for pinned-memory
allocation.

The manager constructs each backend configuration from a concrete saver
configuration. Regular saves use an asynchronous saver with
pinned-memory
staging and a checkpoint barrier. Load-only runs use a synchronous saver
with
no barrier. The final write uses a fresh synchronous saver with a
barrier so it
must complete before the process exits. Building each mode directly
avoids
copying and incrementally rewriting nested dataclass configurations.

Backend storage is a defaulted `__init__` parameter rather than a
`Config`
field. `Configurable.Config` is Tyro-parsed, and a backend storage
object is
not a command-line option; callers needing remote storage pass it
programmatically. The backend config receives the same storage object,
so
saves, loads, and path probes agree.

Retention reuses the shared `purge_thread` worker from
`checkpointer/base.py`,
draining through the backend `Storage` abstraction rather than the
filesystem
directly, and matches the same `step-(\d+)` names the DCP manager
purges.

`last_save_in_hf` is rejected in `Config.__post_init__`; HF export lands
in a
later change and would otherwise fail deep in the backend.

This adapts the implementation to the current OSS layout:
`components/torch_checkpointing_manager.py` is now
`components/checkpointer/torch_checkpointing.py`,
`CheckpointManagerConfig` is
`BaseCheckpointManager.Config`, `purge_worker` is `purge_thread`, and
`LRSchedulersContainer` comes from `components.optimizer`. The earlier
version
also carried its own `if not self.enable` guard in every public method;
those
are gone, since the base class now owns that check and dispatches to the
`_save` / `_wait_for_saving` / `_maybe_wait_for_staging` / `_close`
hooks.

`_should_save` and `_create_checkpoint_id` move to the base because they
depend
only on fields already declared by `BaseCheckpointManager.Config`.
Retention,
step discovery, state selection, and the final-step payload stay
per-manager
because each backend reaches storage differently.

Test Plan:
`pytest tests/unit_tests/cpu/test_torch_checkpointing.py`: 23 passed.

The tests cover save cadence and future tracking, prewarm running once
before
the first scheduled save, load-only selecting a synchronous barrier-free
backend, load-only never constructing a barrier, staging waits through
the
backend lock, configured save timeouts, cleanup after failed saves,
retention,
and the final synchronous model-only save.

Also verified:
- The required backend APIs exist in `torch_checkpointing` 0.1.0.
- `TorchCheckpointingManager.__abstractmethods__` is empty.
- `ufmt` and `flake8 --config=.flake8` pass on both changed files.

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/4188).
* pytorch#4457
* pytorch#4277
* pytorch#4279
* pytorch#4278
* pytorch#4191
* pytorch#4190
* pytorch#4189
* __->__ pytorch#4188

For some reason, `varlen` is routed to FA3 on Hopper GPUs. Is there a
particular reason for this?

According to the [FlashAttention
repository](https://github.com/dao-ailab/flash-attention?), FA4 is
optimized for Hopper and Blackwell GPUs.

This PR routes Hopper to FA4 too

---
# UPDATE

I am repurposing this PR to add a small comment about this, so that
others do not fall for the same mistake. If you think it is too much,
you can simply close this.

Sorry for the mess here!

## Summary
Docs-only cleanup of stale model notes. No invented parity/MFU numbers.

- GPT-OSS README: CP/PP are already covered by model-suite jobs; drop
the stale TODO and note the FlexAttention zero-bubble caveat
- Qwen3 README: remove "under development", fix the aux-loss-free claim
to match the original recipe (aux loss), and list missing
`config_registry` sizes
- Root README: replace helper-script bullets that do not exist with
`download_hf_assets.py`, `scripts/checkpoint_conversion/`, and
`scripts/loss_compare.py`
- Add `torchtitan/models/llama3/README.md` (tokenizer, run commands,
parallelism table). Addresses the Llama 3 slice of pytorch#2354
- Flux README: mark `get_nparams_and_flops()` as done

## Test plan
- [x] Cross-checked recipes/tests mentioned in the new text
(`gpt_oss_pp+fsdp+cp+ep+sacop`, `sft_qwen3_8b_math`, `llama3_8b_mxfp8`,
Flux FLOPs helper)
- [ ] Docs review only

Introducing `with subgraph(name, role)` that allows to outline parts of
the graph into standalone invoke_subgrah() regions, that will be
compiled independently by inductor, keeping them in the same graph.


Example from the test:
```
    def test_subgraph_contextmanager_example_outlines_region(self):
        def f(x):
            with subgraph("chunk_0", role="loss_chunk"):
                a = torch.ops.aten.sin.default(x)
                b = torch.ops.aten.cos.default(a)
            return torch.ops.aten.add.Tensor(b, x)
 ```

This allows to express for example "chunking", when we do not want inductor to fuse/reuse across chunk iterations,
while having them in one big graph.

This allows then to do optimizations across those subgraphs.

E.g. CSE on chunk subgraphs to extract the same fsdp-unshard.

To match dynamo-aotautograd peak memory we have to also add features that exist in aotautograd:
1. applying mincut between forward and backward of the hops - (to have the same save/recompute logic), this was important for recomputes of fp32 casts.
2. Using AotAutograd decompositions before applying mincut, which allows for mincut to save decomposed edges.

## Summary

This PR overhauls MXFP8 linear training in TorchTitan. The key design
choice is a narrow TorchAO/TorchTitan boundary: **TorchTitan relies on
TorchAO only for MXFP8 quantization and layout kernels, while TorchTitan
owns autograd, FSDP cache management, and model policy.** Thanks to
Vasiliy Kuznetsov for adding the TorchAO kernel support used here.

The main changes are:

- FSDP owns the quantized weights and scales, so their lifetime follows
the normal FSDP unshard/reshard lifecycle and composes with both values
of `reshard_after_forward`.
- Weight quantization is pluggable. The built-in strategy uses 2D
`32x32` scaling for weights, while inputs use standard 1D scaling.
- `MXFP8Linear` can save either the original BF16 input or its
columnwise-quantized representation for WGRAD. TorchTitan model configs
select the optimized policy only where the BF16 input is not retained
elsewhere.

## TorchAO and TorchTitan boundary

TorchTitan currently uses the following kernel-level operations from
TorchAO:

- `mxfp8_quantize_cuda` for rowwise and columnwise activation
quantization.
- `triton_to_mxfp8_32x32_swizzle_dim0_and_dim1` for 2D weight
quantization and Blackwell-ready layouts.
- `triton_mx_block_rearrange` for scale layout conversion.

TorchTitan owns everything that interacts with training structure:

- `_MXFP8LinearFunction` and `MXFP8Linear`.
- FSDP tensor subclasses and pre/post-all-gather hooks.
- Quantized-weight storage and lifetime.
- The pluggable weight strategy.
- Model-specific activation-storage policy.

The current kernels are sufficient for this integration. More heavily
fused kernels can be added later as small, independent TorchAO changes
without moving FSDP or parallelism policy into TorchAO.

## FSDP-managed quantized weights

The FSDP post-all-gather hook quantizes the unsharded BF16 weight and
returns the independent MXFP8 qdata and scale tensors for FSDP to
manage. There is no separate module-level or autograd-level weight
cache.

This gives the quantized representation the same lifecycle as the
unsharded FSDP parameter:

| `reshard_after_forward` | Behavior |
|---|---|
| `False` | Quantize on the first unshard and reuse the MXFP8 operands
across PP microbatches. PP can retain them through the microbatch
sequence and release them after the final backward. |
| `True` | Reshard after forward. Backward performs another BF16
all-gather and post-all-gather quantization. |

FSDP keeps the compute-weight object stable and allocates, frees, or
refills its inner tensors through the post-all-gather `out=` contract.
This lets us reuse the existing FSDP state machine instead of
maintaining a second cache lifecycle in `MXFP8Linear`.

The implementation also uses the tensor-subclass API from
[pytorch/pytorch#194114](pytorch/pytorch#194114),
which allows FSDP to release the BF16 all-gather output after the
independent MXFP8 representation has been constructed.

## Pluggable quantization strategy

The practical default used by this PR is:

| Operand | Scaling |
|---|---|
| Weights | 2D `32x32` |
| FPROP inputs | 1D `1x32` |
| WGRAD inputs | 1D `32x1` |

There are two reasons for this choice:

1. With 1D weight scaling, the scale groups needed by FPROP and DGRAD
have different orientations. The two weight representations must
therefore be quantized separately and may contain different quantized
values. With square `32x32` tiles, the quantization groups are invariant
under transpose, so DGRAD qdata can be a transpose view of the FPROP
qdata.
2. Inputs remain one-dimensional because a 2D activation scale may span
multiple sequence positions. In causal attention, that would allow a
token's quantization scale to depend on future tokens.

The PR intentionally provides only the built-in 2D weight strategy. The
implementation remains extensible through
`MXFP8WeightQuantizationStrategy`: a different strategy supplies its
quantization implementation, and strategies with aliased storage can
additionally describe how their logical GEMM operands are reconstructed
from the independent tensors managed by FSDP.

Every strategy must return qdata and blocked scales compatible with the
`scaled_mm` scaling and swizzle contract used by `MXFP8Linear`.

## Revamped `MXFP8Linear`

The previous activation path was:

```text
Forward:  rowwise-quantize X + save BF16 X
Backward: retrieve BF16 X + columnwise-quantize X for WGRAD
```

This remains the conservative default, but the PR adds an optimized
policy:

```text
Forward:  rowwise- and columnwise-quantize X + save columnwise MXFP8 X
Backward: retrieve the saved columnwise MXFP8 X
```

| `wgrad_input_storage` | Forward quantization | Saved for WGRAD |
Backward work |
|---|---|---|---|
| `high_precision` (default) | Rowwise only | Original BF16 `X` |
Columnwise-quantize `X` |
| `quantized` | Fused rowwise + columnwise | Columnwise qdata and scales
| Reuse saved operand |

The quantized policy can reduce saved-activation memory and avoids a
quantization pass during backward. It also allows the rowwise and
columnwise outputs to be produced in one kernel call, reducing HBM
traffic. This design is similar to the activation handling used by
NVIDIA Transformer Engine.

The important caveat is that this saves memory only when no other
operation retains the same BF16 `X`. If BF16 `X` is already retained
elsewhere, saving an additional quantized representation increases
memory. We therefore keep `high_precision` as the default to avoid an
unexpected memory regression or OOM, and select `quantized` explicitly
for audited modules.

| Is the same BF16 input retained elsewhere? | Policy | Reason |
|---|---|---|
| Yes or unknown | `high_precision` | Avoid keeping both BF16 and
quantized WGRAD inputs. |
| No | `quantized` | Replace the saved BF16 WGRAD input with the smaller
quantized representation. |

The current TorchTitan policies are:

| Model | Modules using `quantized` |
|---|---|
| Llama 3 | `attention.qkv_linear.wqkv`, `feed_forward.w2` |
| DeepSeek V3 | `attention.wkv_b`, `feed_forward.w2`,
`shared_experts.w2` |
| Flux | None; it retains the conservative default |

All other converted linears use `high_precision`. The standard Trainer
and GraphTrainer share these model policies. The grouped-expert/Grouped
GEMM path is unchanged by this PR.

## Testing

- Unit coverage for activation policies, pluggable weight strategies,
and FSDP tensor reconstruction.
- FSDP lifecycle coverage for both values of `reshard_after_forward`,
including PP cache reuse and final-backward release.
- Compiled FSDP, PP, and GraphTrainer training runs.

## Appendix: RAF and activation-checkpointing trade-offs

Activation checkpointing affects which weight representations are needed
during the backward unshard. It does not change the WGRAD input-storage
policy described above.

The current post-all-gather hook constructs both FPROP and DGRAD weight
operands on every actual unshard:

| RAF | AC | Current behavior | Ideal behavior |
|---|---|---|---|
| `False` | Off | Construct both once and reuse | Construct both once
and reuse |
| `False` | On | Construct both once and reuse | Construct both once and
reuse |
| `True` | Off | Construct both in forward and both again in backward |
FPROP in forward, DGRAD in backward |
| `True` | On | Construct both in forward and both again before backward
| FPROP in forward, both before backward |

For `RAF=False`, the current behavior is already ideal because both
representations are constructed once and reused.

For `RAF=True`, the initial forward only needs FPROP. Without AC, the
backward unshard only needs DGRAD. With AC, however, the backward
unshard must support both the recomputed forward and DGRAD.

FSDP internally knows its training phase and RAF setting, but this
context is not currently exposed through the post-all-gather hook. In
addition, checkpoint recomputation is not always known when the backward
unshard begins. This PR deliberately keeps the hook simple and correct
by constructing both representations. A future FSDP hook extension could
expose the unshard context and enable phase-specific construction.

… -> renderers.Renderer

Address the pytorch#4444 review (tianyu-l, wwwjn, HosseinKaviani-H).

- `RendererConfig(Configurable.Config)` is the renderer slot on `Controller.Config`;
  `build(*, tokenizer)` returns the `renderers` protocol object. `RenderersLibraryConfig`
  holds the library's typed config as `renderers_config`, serializes it locally
  (`model_dump(mode="json")`) and calls `create_renderer`. `MuseGlimmerRendererConfig` is a
  plain dataclass with its own `build`; `TorchTitanRendererConfig`, `renderer_cls` and the
  `isinstance` dispatch in `build_renderer` are gone.
- `Controller.Config.tokenizer: HuggingFaceTokenizer.Config`; tokenizer and renderer are
  built from config in the controller, the rollout workers (`setup_async` carries
  `tokenizer_config`) and `generate.py`.
- `torchtitan/config/configurable.py` reverted to main; no core change.
- `RendererTokenizer` -> `RendererTokenizerWrapper`.
- `TokenEnv.step` passes the env's `tools` to `parse_response`, as it already did for
  `render_ids` and `bridge_to_next_turn`.
- Recipes: `renderer=RenderersLibraryConfig(renderers_config=Qwen3RendererConfig(...))`;
  the `--renderer.*` CLI flags are gone (renderer options are recipe-owned; every CI recipe
  already sets `enable_thinking=False`).

Validation: alphabet sort Qwen3-0.6B, 2 GPUs, 30 steps, validation reward 0.181 -> 0.526.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…n build

The dataclass config replaced a pydantic one, so bad values were accepted silently:
`thinking_retention="everything"` bridged as "tool_cycle", and a string in either bool knob
was truthy (`answer_from_reasoning_fallback="false"` promoted reasoning to the answer).
`__post_init__` now rejects them. `build` passes `replace(self)` so a recipe object edited
after build cannot desynchronize full renders from bridging (the renderer caches its bridge
policy at construction but reads `retain_reasoning_in_history` live).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>